Papers with auto-regressive language models
A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing sampling algorithms for auto-regressive language models have similar performance . entropy reduction, order preservation and slope preservation are common properties of existing methods . |
| Approach: | They investigate the quality-diversity trade-off between ancestral sampling algorithms for auto-regressive language models. |
| Outcome: | The proposed methods have similar performance to existing methods for open-ended language generation. |
Optimizing Packing and Shuffling Strategies for Enhanced Performance in Generative Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Documents are typically concatenated to chunks of maximum sequence length (MSL) and shuffled in chunks (atom-size chunks). |
| Approach: | They propose to pack and shuff documents in chunks of tokens to prevent overfitting . they also propose to use padding to only include one document per chunk . |
| Outcome: | The proposed method reduces the risk of overfitting and improves generalizability. |
SenticNet 7: A Commonsense-based Neurosymbolic AI Framework for Explainable Sentiment Analysis (2022.lrec-1)
Copied to clipboard
| Challenge: | Despite recent advances, AI still struggles with complex tasks that require commonsense reasoning such as natural language understanding. |
| Approach: | They propose a commonsense-based framework that aims to overcome these limitations in the context of sentiment analysis. |
| Outcome: | The proposed framework overcomes these limitations in the context of sentiment analysis. |
Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Exposure bias is a central problem for auto-regressive language models (LM) it is believed that teacher forcing would cause test-time generation to be incrementally distorted due to the training-generation discrepancy. |
| Approach: | They propose to quantify the impact of exposure bias in quality, diversity, consistency and consistency by using ground-truth data prefixes instead of prefix generated by the model. |
| Outcome: | The proposed model performs better when the training-generation discrepancy is removed . the model is more robust and self-recovery ability is shown to counter exposure bias. |
NameGuess: Column Name Expansion for Tabular Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Tabular data is used for storing and organizing information in web and enterprise applications. |
| Approach: | They propose a task to expand column names as a natural language generation problem by conditioning on table content and column header names to improve auto-regressive models. |
| Outcome: | The proposed task improves auto-regressive models on table content and column header names to match human performance. |
Diffusion Guided Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing guidance methods for text generation are prone to decoding errors and degrade performance. |
| Approach: | They propose a model that steers an auto-regressive language model to generate text with desired properties. |
| Outcome: | The proposed model outperforms existing guidance methods on a wide range of benchmark data sets. |